GAO 



United States Government Accountability Office 



Report to Congressional Requesters 



IMMIGRATION 
SERVICES 

Better Contracting 
Practices Needed at 
Call Centers 




GAO-05-526 



June 2005 



G A O 



Accountability* Integrity* Reliability 

Highlights 

Highlights of GAO-05-526, a report to 
congressional requesters 



IMMIGRATION SERVICES 

Better Contracting Practices 
Needed at Call Centers 



Why GAO Did This Study 

The U.S. Citizenship and 
Immigration Services (USCIS) 
bureau within the Department of 
Homeland Security (DHS) provides 
toll-free telephone assistance 
through call centers to immigrants, 
their attorneys, and others seeking 
information about U.S. immigration 
services and benefits. As the 
volume of calls increased — from 
about 13 million calls in fiscal year 
2002 to about 21 million calls in 
fiscal year 2004 — questions were 
raised about USCIS's ability to 
ensure the reliability and accuracy 
of the information provided at call 
centers run by an independent 
contractor. 

This report analyzes: (1) the 
performance measures established 
by USCIS to monitor and evaluate 
the performance of contractor- 
operated call centers; (2) how 
performance measures were used 
to evaluate the contractor's 
performance; and (3) any actions 
USCIS has taken, or plans to take, 
to strengthen call center 
operations. 



What GAO Recommends 



To improve USCIS's evaluation of 
contractor performance, GAO 
recommends that USCIS take steps 
to ensure that performance 
measurement provisions are 
finalized before awarding new 
contracts and that performance 
evaluation records are properly 
maintained. DHS generally agreed 
with GAO's recommendations and 
indicated USCIS was taking steps 
to implement them. 
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To view the full product, including the scope 
and methodology, click on the link above. 
For more information, contact Paul Jones at 
(202) 512-8777 orjonespl@gao.gov. 



What GAO Found 

USCIS developed seven performance measures intended to assess the 
performance and overall quality of responses provided by customer service 
representatives at contractor-operated call centers. These measures include 
how quickly calls were answered and the accuracy of information provided. 
The contract between USCIS and its contractor stipulated that the 
contractor could earn financial incentive awards if the average monthly 
performance met or exceeded the standards on a quarterly basis at each of 
four call centers. Conversely, financial deductions could be made if the 
standards were not met. 

USCIS did not finalize the terms regarding how the contractor's actual 
performance would be calculated, or scored, before awarding the contract. 
This limited USCIS's ability to exercise performance incentives (positive or 
negative) because the parties could not reach agreement on performance 
terms. USCIS suspended the use of financial incentives while the parties 
negotiated the issue. Agreement was not reached after 16 months, however, 
USCIS determined that the contractor had failed to meet standards for 4 of 
the 7 performance measures in the fourth quarter of 2004 and took action to 
reduce its payments for services. The contractor objected, citing the lack of 
agreement on the performance measurements and the impact of workload 
increases, but USCIS disagreed and stated it would reduce payment. In a 
separate but related matter, USCIS failed to meet contractual, regulatory, 
and GAO standards pertaining to how the contractor's performance would 
be documented — especially with respect to any deficiencies. Finally, USCIS 
exercised its option to extend the call center contract through May 2006, to 
allow time to solicit and award new call center contracts. USCIS said it 
intends to finalize performance measurement terms in the new contracts. 

USCIS used contractor performance data it collected over the course of the 
contract to identify opportunities to improve customer service and call flow, 
among other things. Several initiatives were launched as a result. 

USCIS Call Flow and Call Volume for Fiscal Year 2004 



21.1 million calls 




Source: Developed by GAO from USCIS data. 
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^ G A O 

^ Accountability * Integrity * Reliability 

United States Government Accountability Office 
Washington, DC 20548 



June 30, 2005 
Congressional Requesters: 

The U.S. Citizenship and Immigration Services (USCIS) bureau within the 
Department of Homeland Security (DHS) provides telephone assistance to 
customers calling about U.S. immigration services and benefits. USCIS 
customers — immigrants and their attorneys or family members, 
employers, and others — can call toll-free for information on such topics as 
how to become U.S. citizens, obtain work visas, initiate the naturalization 
process, obtain legal forms, and determine the status of applications for 
benefits. This telephone assistance service helps customers avoid making 
visits to USCIS offices, which could require customers to take time off 
from work and travel long distances. Telephone calls for assistance to 
USCIS have increased in recent years, from about 13 million in fiscal year 
2002 to about 21 million in fiscal year 2004. These calls are handled in a 
variety of ways: by an interactive voice response system; 1 by four call 
centers managed by a private contractor (Pearson Government Solutions) 
engaged by USCIS; and by two call centers operated directly by USCIS. 

You expressed interest in how USCIS monitors and evaluates contractor- 
operated call centers to ensure that they operate reliably and provide 
accurate information to their growing number of customers. This report 
addresses the following: (1) What performance measures did USCIS 
establish to monitor and evaluate the performance of contractor-operated 
call centers? (2) How were these performance measures used to evaluate 
the contractor's performance? (3) What actions, if any, did USCIS take or 
plan to take to strengthen call center operations? 

To address these objectives, we obtained and analyzed information from 
USCIS officials in Washington, D.C., from contractor officials in Arlington, 
Virginia, and at one of four contractor-operated call centers. 2 At these 
locations, we interviewed officials and collected and analyzed pertinent 
documentation, including descriptions of monitoring and evaluation 
programs, contract requirements for call center operations, and 
summaries of call center evaluation results. We assessed the reliability of 



1 An automated voice system with a touch-tone menu. 

2 The contractor considers the locations of its call centers to be proprietary information. 
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telephone call volume data provided to USCIS by a telecommunications 
vendor, as well as USCIS and contractor staffing data. To carry out our 
data reliability assessments, we (1) reviewed information about the data, 
systems that produced the data, and data quality control procedures, and 
(2) interviewed USCIS and contractor officials knowledgeable about the 
data as necessary. We determined that these data were sufficiently reliable 
for purposes of this report. 

We conducted our work between May 2004 and May 2005 in accordance 
with generally accepted auditing standards. (See app. I for details about 
our scope and methodology.) 



ReSllltS in Brief USCIS developed seven performance measures that were intended to 

assess the performance of and overall quality of responses provided by 
customer service representatives at contractor-operated call centers. 
These measures were described in a contract signed by both parties in 
January 2002. Two performance measures, for example, were to assess the 
accuracy of information provided to callers and how quickly calls were 
answered. Based on provisions in the contract, the contractor could earn 
financial incentive awards if certain standards or goals set for each 
performance measure were met or exceeded on a quarterly basis at each 
of the call centers — for instance, if callers waited an average of 30 seconds 
to 36 seconds for their calls to be answered. Conversely, the contract 
provided for negative financial incentives (penalties), whereby USCIS 
could deduct from payments owed to the contractor on a quarterly basis if 
the standards were not met. 



USCIS collected data on the contractor's performance on a monthly basis 
since June 2002, when the contract was awarded. However, these data 
were not used for the purpose of applying financial incentives (either 
positive or negative) until the fourth quarter of 2004. The use of financial 
incentives was suspended because USCIS awarded the contract without 
finalizing how the contractor's actual performance would be calculated, or 
scored, for the various performance measures. The two parties negotiated 
on this issue over a period of about 16 months after the contract was 
awarded before abandoning the effort. Thereafter, USCIS determined that, 
for the fourth quarter of 2004, the contractor had failed to meet standards 
for four of the seven performance measures, and announced its intention 
to reduce payments to the contractor for services. The contractor objected 
to USCIS's decision on the grounds that the performance measurements 
had not been finalized and that changes in call center workloads affected 
the basis for applying financial incentives. USCIS disagreed and stated it 
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would reduce payment to the contractor. Although this dispute had not 
been resolved, USCIS exercised its option to extend the current call center 
contract through May 2006 to allow time to solicit and award new call 
center contracts. The exercise of this option has no effect on the 
contract's performance measurement terms, which is the source of the 
parties' disagreement. In a separate but related matter, USCIS failed to 
meet contractual, regulatory, and GAO standards pertaining to how the 
contractor's performance was documented — especially with respect to 
any deficiencies. For example, USCIS's failure to obtain the contractor's 
written acknowledgement of USCIS-identified performance deficiencies 
did not meet the notification procedures established by the contract, 
documentation requirements of the Federal Acquisition Regulations, and 
GAO's standards for internal control. Consequently, USCIS did not 
maintain a complete and reliable record of the contractor's performance 
needed to ensure accountability. 

USCIS used the contractor performance data it collected monthly over the 
course of the contract to identify opportunities to improve customer 
service, help better respond to customer inquiries, and manage the flow of 
calls into call centers. Several initiatives have been launched as a result. 
For example, USCIS has implemented new call-routing procedures 
designed, in part, to route customers' calls to service representatives more 
quickly and reduce delays (on-hold time) that arise when call volume is 
heavy. These and other initiatives were undertaken by USCIS concurrent 
with its efforts to negotiate contract differences with the contractor. It is 
too early to assess the impact of these initiatives. 

To improve USCIS's efforts for evaluating contractor performance and 
encourage quality services at call centers, we recommend that the 
Secretary of Homeland Security require the Director of USCIS take the 
following two actions: (1) finalize contract terms related to specific 
performance measurement requirements, before awarding new 
performance-based call center contracts; and (2) maintain readily 
available written records of performance assessments and performance 
evaluation meetings with the contractor. 

DHS and the contractor provided formal comments and technical 
comments on a draft of this report, which we have incorporated, as 
appropriate. DHS generally agreed with our recommendations. The 
contractor said the report accurately summarizes the complex nature of 
CIS's call center program and several challenges created by significant 
post-award changes to that program. 
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Background The Homeland Security Act of 2002 established USCIS within DHS. 3 USCIS 

^ is responsible for several functions transferred on March 1, 2003, from the 

former Immigration Services Division of the Immigration and 
Naturalization Service (INS) under the Department of Justice. 4 These 
functions include providing services or benefits to facilitate entry, 
residence, employment, and naturalization of legal immigrants; processing 
applications for U.S. citizenship/naturalization; and rendering decisions 
about immigration-related matters. 



Call Center Operations The USCIS Information & Customer Service Division is responsible for 

operating the National Customer Service Center (NCSC), which was 
established in 1997 to provide nationwide assistance by telephone to 
customers calling about immigration services and benefits. When a 
customer calls the NCSC toll-free number (1-800-375-5283), the call is 
received by the interactive voice response system. The system features 
automated, self-service options 24-hours a day, 7 days a week. If the 
system cannot address a customer's concerns or needs or if a customer 
requests live assistance, then the call is generally routed to one of the four 
NCSC contract call centers, known as Tier l. 5 These four centers are 
operated by the contractor, Pearson. If a question posed by a customer is 
particularly complex or otherwise cannot be answered at the Tier 1 level, 
the call is transferred to one of the two USCIS-operated call centers, 
known as Tier 2. 6 Figure 1 shows the organization of NCSC, including the 
call centers. 



3 Pub. L. No. 107-296, § 451 (Nov. 25, 2002). 

Throughout this report, we use USCIS to refer to both the new USCIS and the former 
Immigration Services Division of INS. 

5 If a customer attempts to obtain live assistance outside normal business hours, the 
customer is told when to call back to obtain the assistance. 

6 In some cases, customers' calls are routed directly from the interactive voice response 
system to Tier 2, bypassing Tier 1. 
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Figure 1: Organization of USCIS's National Customer Service Center and Call 
Centers 
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Source: Developed by GAO from USCIS data. 



In fiscal year 2004, almost half of 21.1 million calls made to NCSC were 
handled and completed by the interactive voice response system and the 
rest were generally routed to Tier 1. Customer service representatives 
(CSR) at Tier 1 respond to inquiries in English or Spanish. The CSRs focus 
primarily on providing administrative information to customers by using a 
series of scripts provided by USCIS. For example, if a customer needs 
what USCIS considers basic information, such as USCIS local offices' 
hours of operations, eligibility requirements, and procedures to follow, 
such questions are to be answered by CSRs at Tier 1 call centers using 
specific scripts. In addition, CSRs are to refer customers to USCIS service 
centers and local offices, for such things as changes of address and 
appointment scheduling at USCIS application support centers. (Some of 
these tasks may alternatively be performed by customers through the 
USCIS Web site — www.uscis.gov.) As of April 2005, the four Tier 1 call 
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centers employed over 450 CSRs. 7 Figure 2 shows CSRs processing calls at 
a Tier 1 call center. 



Figure 2: Tier 1 Customer Service Representatives Processing Immigration-Related Calls at a Contractor-Operated Call 
Center 




Source: USCIS contractor. 



At the two USCIS-operated Tier 2 call centers, calls are handled by 
immigration information officers (IIO) — immigration specialists with in- 
depth knowledge of immigration laws, non-immigrant visas, naturalization, 
asylum and refugee status, and other related policies and procedures. As 
of April 2005, the Tier 2 call centers operated by USCIS had 111 IIOs. 
About 5 percent, or about 590,000, of the calls going to Tier 1 CSRs were 



7 The contractor considers the precise number of CSRs at its call centers and their locations 
to be proprietary information. 
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rerouted to Tier 2 IIOs in fiscal year 2004. Figure 3 shows the call volume 
handled by the interactive voice response system, Tier 1 call centers, and 
Tier 2 call centers during fiscal year 2004. 



Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004 
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Source: Developed by GAO from USCIS data. 

a "Completed calls" refer to calls that have ended, not necessarily to calls where customers' questions 
or issues have been resolved. 



Performance-Based In January 2002, USCIS awarded a performance-based service contract for 

Service Contract the management of four Tier 1 call centers. 8 In making this award, USCIS 

obtained acquisition services from the Department of Veterans Affairs 
(VA), and the contracting officer who signed and was responsible for 
administering the contract was a VA employee working on behalf of 
USCIS. 9 The contracting officer's technical representative (COTR), a 
USCIS employee, was also responsible for administering the contract. In 
commenting on a draft of this report, DHS noted that by agreement of both 
the VA and USCIS, on April 20, 2005, USCIS assumed responsibility for 
administering the contract. The contract was awarded for a base year, 



The term "contract" in this report refers specifically to "Delivery Order No. 591," which 
was awarded to the contractor following a competitive procedure through the General 
Services Administration's Federal Supply Schedule. 

9 The contract was awarded before DHS was established while what is now USCIS was part 
of INS. 
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beginning on June 1, 2002, plus 4 option years (1-year renewable 
extensions of the contract, three of which have been exercised as of June 
2005). Through calendar year 2004, USCIS paid $64.6 million to the 
contractor for the Tier 1 call center operations. 

According to the Office of Federal Procurement Policy (OFPP) in the 
Office of Management and Budget, performance-based service contracts 
are designed to focus on results. 10 Their purpose is to ensure that 
contractors are given the freedom to determine how to meet the 
government's performance objectives, that appropriate quality levels are 
achieved, and that payment is made only for services that meet these 
levels. This type of contract is to emphasize standards for customer 
service and measurement of performance and may offer financial 
incentives, both positive and negative, to encourage quality performance. 
According to OFPP, call centers are suited to this type of contract 
because, among other things, they emphasize achieving results by meeting 
customer service standards. According to OFPP, with performance-based 
service contracts, incentive payments made to an independent contractor 
are to be contingent on the contractor's ability to meet the government's 
performance standards; the contract does not specify how those standards 
are to be met. Thus, the contractor retains discretion in determining how 
to meet performance standards specified in the contract, for example, how 
many CSRs to hire to ensure calls are answered within a contractually 
specified time. Other elements suggested for using a performance-based 
service contract include (1) identifying the agency's needs and addressing 
those needs with performance requirements that describe required service 
results; (2) establishing performance standards that describe the required 
performance level; and (3) establishing a quality assurance plan for 
assessing contractor performance in order to ensure that the contractor 
has performed in accordance with the standards. 



OFPP and the National Performance Review coordinated a federal interagency working 
group to develop generic performance-based concepts for government agencies to use in 
the procurement of call center services. The group developed a reference source to help 
program and acquisition officials draft performance requirements and standards related to 
call center performance-based contracts: Performance Based Concepts for Telephone Call 
Center Contracting, Office of Federal Procurement Policy (May 27, 1997). 
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USCIS used a multi-faceted approach to monitor and evaluate the quality 
of information and service provided by CSRs to customers calling 
contractor-operated Tier 1 call centers. 11 This approach used seven 
performance measures. USCIS obtained performance data from the 
contractor's monitoring of selected telephone calls; customer satisfaction 
surveys; and a telecommunications vendor (telephone company). 12 In 
addition, USCIS used an independent consulting firm to monitor CSRs' 
telephone calls and conduct a "mystery shopper" program assessing CSRs' 
responses to customers. 



USCIS Set Call Center In order to monitor and evaluate the performance of the four contractor- 

Performance Measures for operated Tier 1 call centers, USCIS planned to use seven performance 
the Contractor to Meet measures. These measures were to evaluate the quality of customers' 

telephone interactions with CSRs; the accuracy of information provided to 
callers over the telephone; the accuracy of callers' information recorded 
by CSRs; callers' levels of satisfaction; how quickly CSRs handled calls 
(two measures); and the number of calls abandoned by customers put on 
hold. According to USCIS officials, USCIS established the performance 
measures based on a review of industry standards for both government 
and private-sector call center operations. The measures were described in 
a section of the contract called the Performance Requirements Summary 
(PRS). 

Under the PRS, these performance measures comprised one of three 
components upon which the contractor's performance score was based. 
The other two components were the standard, or goal, set for each 
measure, identifying the performance levels the contractor was expected 
to meet (e.g., callers will wait an average of 30 to 36 seconds before their 
calls are answered), and the performance calculation that USCIS would 
use to analyze performance data (e.g., total delay of all calls divided by the 
total number of calls). 13 The PRS listing of the seven performance 
measures included a "sample calculation" for each of the measures, and 



USCIS monitors and evaluates operations at both Tier 1 and Tier 2 call centers. This 
report focuses on USCIS's evaluation of Tier 1 contractor-operated call centers. 

12 This telecommunications vendor is also responsible for providing the interactive voice 
response system and routing customers' telephone calls from this system to Tier 1 and Tier 
2 call centers. 

13 In this report, we collectively refer to these three components as "performance 
measurement requirements." 



USCIS Established 
Performance 
Measures to Assess 
Contractor's Call 
Center Performance 
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stated that "actual calculations [are] to be determined during Contract 
negotiations." 14 USCIS officials said they intended to negotiate and finalize 
the calculations after a 4- to 6-month phase-in period, and the contract was 
awarded with this provisional language. 

As to the performance measures and their related standards or goals, three 
of USCIS's performance measures are call quality monitoring, accuracy 
of information provided, and accuracy of capturing information. Data 
on these measures are to be collected by the contractor's quality 
assurance staff, who are to randomly monitor two calls per day for each 
CSR. (CSRs are not to know when they are being monitored.) The data 
collected are to be reported to USCIS on a monthly basis. Details on these 
three measures follow: 

• Call quality monitoring. Calls are to be monitored by the contractor's 
quality assurance staff to assess the CSRs' "soft skills," that is, their 
ability to interact with customers, establish customer rapport, maintain 
composure during a call, speak with clarity and professionalism, and 
other factors. Call quality monitoring data are to be captured on a 
standardized form. CSR responses for each of nine different "soft 
skills" are scored as percentages, with scores for the most highly 
valued skills, such as "active listening" — that is, whether the CSR was 
deemed to be attentive when listening to the customers — given more 
weight than the scores for other skills. The nine scores (i.e., 
percentages) are combined for a total "soft skills" score, with 100 
percent as the highest possible score. The performance standard stated 
in the PRS for this measure is that all calls monitored achieve an 
average score of 90 percent to 95 percent after the nine "soft skills" 
scores (i.e., percentages) for each call are combined. (See app. II for 
additional details on the criteria and methodology used to determine 
soft skills scores.) 

• Accuracy of information provided. Calls are to be monitored by the 
contractor's quality assurance staff to determine, among other things, 
whether CSRs provided accurate and complete responses. Using a 



14 The "sample" terms of the contract were never finalized, leading to a dispute between 
USCIS and the contractor that remains unresolved (see p. 15). Examples of "sample 
calculations" listed for different measures in the PRS were "total delay of all calls divided 
by total number of calls" for determining the average length of time it takes before calls are 
answered, and "total number of points achieved divided by total number of points available 
for all quality monitoring scores" for determining call quality. One sample calculation was 
listed for each measure. 
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standardized form, the staff score CSRs on five different efforts, such 
as whether the CSR used software tools appropriately and whether, 
when the callers were asked directly, they indicated that their needs 
had been satisfied. The five efforts are scored as percentages, with 
more weight given to the scores for certain efforts, such as "provides 
complete response." The scores are then combined for a total 
"accuracy of information provided" score, with 100 percent as the 
highest possible score. The performance standard stated in the PRS for 
this measure is that all calls monitored achieve an average score of 95 
percent to 97 percent after the five accuracy scores (i.e., percentages) 
for each call are combined. (See app. II for additional details on the 
criteria and methodology used to determine accuracy of information 
provided.) 

• Accuracy of capturing information. Calls are to be monitored by the 
contractor's quality assurance staff to determine, among other things, 
whether CSRs accurately record and verify the callers' information. 
The staff assess this measure by scoring four efforts, including whether 
a referral to a local USCIS service center or local office was completed 
appropriately and correctly. The four efforts are scored as percentages, 
with more weight given to the scores for certain efforts, such as 
"verifies caller's information." The scores are then combined for a total 
"accuracy of information provided" score, with 100 percent as the 
highest possible score. The performance standard stated in the PRS for 
this measure is that all calls monitored achieve an average score of 95 
percent to 97 percent after the four accuracy scores (i.e., percentages) 
for each call are combined. (See app. II for additional details on the 
criteria and methodology used to determine accuracy of capturing 
information.) 

A fourth performance measure of call quality — customer satisfaction — 
was assessed by an independent consulting firm. Customer satisfaction 
surveys were conducted on a monthly basis to determine if customers 
were satisfied with the service that CSRs provided. 15 At least 375 callers 
are to be randomly selected to be interviewed each month from a 
population of 10,000 randomly identified callers who called within the 30 
days prior to the survey. To measure satisfaction with CSRs, customer 
responses to four interview questions about CSRs are compiled, and the 
overall percentage of respondents indicating satisfaction is calculated. The 



15 The customer satisfaction surveys ask about customer satisfaction with the interactive 
voice response system and IIOs, as well as CSRs. 
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performance standard stated in the PRS for this measure is 80 percent to 
85 percent of the customers surveyed indicating overall satisfaction with 
the CSRs' service. (See app. Ill for additional details on the criteria and 
methodology used to determine customer satisfaction.) 

Three other performance measures involve the collection of statistical 
data by the telecommunications vendor for determining how quickly calls 
are answered. The performance measures and standards in the contract 
for assessing how quickly CSRs answered customers' calls are as follows: 

• Service level. The telecommunications vendor under contract with 
USCIS is to collect information on the number of calls answered by 
CSRs in 20 seconds or less, that is, the number of callers who spoke to 
a CSR within 20 seconds after getting through the interactive voice 
response system. The performance standard stated in the PRS for this 
measure involves two factors: half-hour increments and the length of 
time it took CSRs to answer calls. The standard is that for 80 percent to 
85 percent of the half-hour increments measured, 80 percent of the 
calls are to be answered in 20 seconds or less. 

• Average speed of answer. The telecommunications vendor under 
contract with USCIS is to collect information on the length of time it 
takes for CSRs to answer customers' calls after they are routed to Tier 
1 by the interactive voice response system; that is, how long callers are 
on hold before a CSR answers their call. The performance standard 
stated in the PRS for this measure is that, for all calls routed to Tier 1, 
callers will wait an average of 30 seconds to 36 seconds. 

• Abandoned calls. The telecommunications vendor under contract with 
USCIS is to collect information on the number of calls abandoned by 
customers after getting through the interactive voice response system 
and waiting for a CSR to answer, that is, the number of times that 
customers hang up the telephone while waiting for a CSR. The 
performance standard stated in the PRS for this measure involves two 
factors: half-hour increments and how frequently callers abandon their 
calls. The standard is that for 85 percent to 95 percent of the half-hour 
increments measured, 1 percent to 2 percent of the calls are expected 
to be abandoned before a CSR answers. 

The contract stated that the contractor would be eligible to earn financial 
incentive awards if the average monthly performance met or exceeded the 
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standards on a quarterly basis at each call center, and allowed USCIS to 
make deductions from payments to the contractor if the average monthly 
performance fell below the standards. 16 According to the contract, the 
contractor is not eligible for an incentive award for a particular quarter if 
one of the performance standards is not met by one call center, and USCIS 
may make a deduction from payments to the contractor in that case. In 
addition, USCIS may, at its sole option, elect to include or waive financial 
incentives as it deems appropriate. 



USCIS Carried Out 
Additional Quality 
Assurance Efforts 



In addition to the performance data collected by the contractor's own 
quality assurance staff, an independent consulting firm, and the 
telecommunications vendor, USCIS took two additional steps to measure 
call center performance for quality assurance purposes. First, to help 
ensure that the contractor's scoring of call-quality performance measures 
was reliable, USCIS used another independent consulting firm to validate 
the results of the contractor's efforts by monitoring two calls per month 
for each CSR. Data were gathered and provided to USCIS on a monthly 
basis. (See app. IV for additional details on the criteria and methodology 
used by the independent consulting firm to conduct call monitoring.) 

Second, in April 2003, USCIS engaged the same independent consulting 
firm to carry out a "mystery shopper" program to assess the completeness 
and accuracy of CSRs' answers to callers. Under this program, an 
independent consultant places random calls — 1,200 each month — to Tier 1 
call centers using various scripts provided by USCIS. As of April 2005, the 
scripts used in these calls covered 32 different scenarios, or types of calls, 
and 100 new scenarios were being developed. The calls are conducted in 
English and Spanish. (See app. V for an example of a mystery shopper 
scenario.) 



16 Up to 10 percent of payments can be added to or deducted from the contractor's billed 
amounts, depending upon whether or not the performance standards are met. 
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USCIS Evaluated 
Contractor's 
Performance but 
Suspended Use of 
Financial Incentives 
for More Than 2 Years 
Due to Performance 
Measurement Dispute 



USCIS did not reach agreement with the contractor on how to apply the 
performance measurement requirements described in the PRS before 
awarding the performance-based service contract. USCIS suspended all 
financial incentives, positive or negative, while the parties negotiated this 
issue over a period of about 16 months without reaching agreement. After 
negotiations were abandoned, USCIS determined that, for the fourth 
quarter of 2004, the contractor had failed to meet four of seven 
performance measures and merited a payment deduction. The contractor 
disagreed on the grounds that the performance measurements had not 
been finalized and that changes in call center workloads affected the basis 
for applying financial incentives. In a separate matter, USCIS failed to 
ensure that all contractual, regulatory, and GAO standards pertaining to 
the documentation of the contractor's performance were fulfilled. 



USCIS Awarded the Call 
Center Contract without 
Finalizing Performance 
Measurement 
Requirements 



The performance measurement requirements described in the PRS were 
not completely finalized before the contract was awarded. The language 
referring to "sample" calculations for determining how performance would 
be measured remained in the contract after it was in force. In commenting 
on a draft of this report, DHS said that at the time of the contract award, 
USCIS management believed it was appropriate to let the winning vendor 
have some input into the performance measurement methodology since 
this contract represented a transition to performance-based contracting 
for call center operations. 

The negotiations between USCIS and the contractor on this issue began in 
January 2003 (after a phase-in period) and continued intermittently until 
April 2004, when they were abandoned. While negotiations were taking 
place and after they were abandoned, USCIS obtained monthly data 
relating to the contractor's performance on the seven performance 
measures and compared those data to the standards. 17 USCIS considered 
the measures and standards themselves to be nonnegotiable; the 
contractor, on the other hand, considered them as part of the "sample 
calculations" and, thus, negotiable. For over 2 years, USCIS did not use 
any of the resulting performance scores for the purpose of calculating 
financial incentive awards or payment deductions under the contract 
because the terms of the PRS remained unresolved between the parties. 



USCIS started collecting data on the contractor's performance in June 2002, the first 
month of the contractor's operations. We are not presenting the performance data used by 
USCIS because the performance measurement terms of the PRS were unresolved between 
USCIS and the contractor as of April 2005. 
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The contractor maintained that the performance scores were "potential 
scores" and were to be used by the parties in reaching an agreement on 
how to structure the PRS. 

On September 1, 2004, the contracting officer, representing USCIS, sent a 
letter to the contractor advising that USCIS would begin evaluating the 
contractor's performance and determining a financial incentive award or 
payment deduction for the fourth quarter of the calendar year (October 1 
through December 31). USCIS officials told us they decided to take this 
action because they had concluded that negotiations with the contractor 
were unlikely to result in an agreement on the PRS. The contractor 
objected to USCIS's decision to carry out this evaluation. By letter dated 
November 29, 2004, the contractor stated that, under the terms of the 
contract, USCIS could not unilaterally determine the performance 
measurement requirements because all aspects of the requirements were 
negotiable, including the performance standards. 

The contractor further stated that an evaluation of its performance must 
take into account certain changes that took place to the work required 
under the contract. For example, the contractor stated that the number of 
USCIS-provided scripts, containing information for CSRs to address 
callers' inquiries, had grown to more than 2,300 pages from approximately 
400 script pages in June 2002. 18 According to the contractor, these changes 
significantly increased the average amount of time needed to handle a call 
and affected the contractor's ability to meet the performance standards 
imposed by USCIS. According to the contractor, USCIS's unilateral 
imposition of performance measurement requirements that did not 
account for the changed work requirements was inconsistent with Federal 
Acquisition Regulation (FAR) 16.402(g), which provides that "[i]t is 
essential that the Government and contractor agree explicitly on the effect 
that contract changes (e.g., pursuant to the Changes clause) will have on 
performance incentives." 

Nevertheless, by letter dated February 11, 2005, USCIS's contracting 
officer notified the contractor of the evaluation results for the period of 
October through December 2004. The results showed that the contractor 
met the standards for three of the seven performance measures and did 



18 USCIS revised the scripts to provide new and updated immigration-related information on 
a variety of topics. In its technical comments on a draft of this report, DHS said that the 
contractor had been compensated for the increased script content and average call 
duration. 
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not meet the standards for the other four measures. USCIS determined 
that, as a result of this performance, payments due to the contractor for 
services would be reduced. The letter noted that the contractor could 
submit its own data regarding performance during this period. Following 
the review of any data submitted, USCIS would take action to make the 
appropriate payment deduction, waive the payment deduction, or pay an 
appropriate incentive award. 

The contractor requested, by letter dated February 25, 2005, that USCIS 
waive implementation of the financial incentives, both positive and 
negative. The contractor reiterated its position that USCIS's unilateral 
implementation of the performance measurement requirements as 
currently written in the contract, without sufficient regard for substantial 
changes to the contract and the changing nature of the program, was not 
appropriate. The contractor stated that it was ready to resume 
negotiations on this subject so that fair and equitable financial incentives 
would be established. The contractor further stated that it had determined 
the payment deduction was incorrectly calculated by USCIS. 

USCIS's contracting officer responded, by letter dated April 15, 2005, that 
the government would not agree to waive implementation of the financial 
incentives and a deduction would be made from the next payment to the 
contractor. The letter stated that USCIS did not unilaterally create and 
impose the performance measurement requirements, which were included 
in the negotiated contract that USCIS and the contractor agreed to. 
Regarding the contractor's assertion that the average amount of time 
needed to handle calls had significantly increased, the letter noted that the 
performance measurement requirements would apply regardless of the 
average length of calls at any given time. 

According to FAR and OFPP guidance 19 on performance-based service 
contracting, the precise method for measuring performance should have 
been agreed upon between USCIS and the contractor before the contract 
was signed and implemented. FAR § 16.401 states that performance-based 
service contracts should establish "reasonable and attainable 
[performance] targets that are clearly communicated to the contractor." 
According to OFPP, performance measurement techniques (i.e., how 
performance will be assessed to determine whether standards have been 



Performance Based Concepts for Telephone Call Center Contracting, Office of Federal 
Procurement Policy (May 27, 1997). 
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met) are essential elements of performance-based service contracting and 
should be clearly stated. In addition, according to OFPP, performance- 
based service contracts emphasize that all aspects of an acquisition be 
structured around the purpose of the work to be performed, that 
appropriate performance quality levels are achieved, that payment is made 
only for services that meet these levels, and that financial incentives are 
awarded to encourage quality performance. 



USCIS Plans to Solicit New 
Tier 1 Call Center 
Contracts with Changes to 
Improve Performance 



Although the disagreement between the two parties had not been resolved, 
USCIS exercised its option to extend the current call center contract for 
another year through May 31, 2006, to allow time to solicit and award new 
call center contracts. 20 The exercise of this option has no effect on the 
contract's performance measurement terms, which is the source of the 
parties' dispute. USCIS officials said they plan to award new performance- 
based service contracts for Tier 1 operations to two vendors, with the two 
vendors fully operational by June 2006, to improve the handling of 
customers' calls to Tier l. 21 



USCIS officials told us they intend for the new contracts to include certain 
changes meant to improve Tier 1 call center operations and to incorporate 
OFPP guidance on performance-based contracting. USCIS officials told us 
that, unlike the current contract, the new PRS will clearly specify how 
contractor performance will be assessed and will not leave any terms open 
for post-award negotiation. In addition, USCIS officials said the new 
contracts will include independent call monitoring and the mystery 
shopper program as performance measurement tools to assess the quality 
of the Tier 1 CSRs' responses to customers, including the accuracy and 
reliability of the information provided. At the time of our review, USCIS 
officials said that the solicitation was going through DHS's contract review 
process and DHS had not issued the solicitation for a new contract 
containing these changes. DHS said in its comments on a draft of this 
report that the solicitation was with the DHS Procurement Office for 
review and issuance. 



20 USCIS officials said that DHS will conduct the acquisition and USCIS will administer the 
contract, without seeking inter-agency assistance from the VA as it previously did with the 
current call center contract. 

21 The current contractor may compete for the new contracts. 
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USCIS and Contractor Did 
Not Fully Document 
Quality Assurance 
Activities 



As part of its quality assurance responsibilities under the current contract, 
USCIS is to keep written records of observations about the contractor's 
performance based on periodic evaluations comparing performance data 
to standards in the PRS. USCIS's contracting officer's technical 
representative (COTR), who is responsible for administering the contract, 
is to use these written observations to notify the contractor if there are 
deficiencies — specifically, if the contractor does not meet the performance 
standards. The contractor is required to sign and date such observations to 
acknowledge that the COTR apprised it of any deficiencies. 

USCIS and contractor officials said they met at least quarterly (monthly, 
since October 2004) to discuss performance, performance data, and other 
items. USCIS officials said they provided the contractor with 
documentation containing performance and other data to discuss at these 
meetings. USCIS officials said some of this documentation identified 
performance deficiencies. However, contractor officials said they viewed 
the performance data as "potential scores" to be considered during 
negotiations. To the extent that USCIS considered the performance data as 
notification of deficiencies, it did not follow contractual procedures 
requiring the COTR to obtain the contractor's signature acknowledging 
notification of the deficiencies. In addition, neither USCIS nor the 
contractor kept minutes of these meetings. 

According to FAR § 46.104(c), the government should maintain, as part of 
the performance records of a contract, suitable records reflecting the 
nature of its contract quality assurance actions. With respect to any 
performance deficiencies, the government's records should include, 
among other things, the number and type of defects observed and any 
actions to correct deficiencies. Further, according to GAO's standards for 
internal control in the federal government, for an agency to run and 
control its operations, it must have relevant, reliable information relating 
to internal events. 22 All transactions and other significant events need to be 
clearly documented, and the documentation should be readily available for 
examination. This information should be recorded and communicated to 
management and others within the agency who need it to carry out their 
responsibilities. GAO's standards provide a framework for establishing 
and maintaining internal control and for identifying and addressing major 



GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-2 1.3.1 
(Washington, D.C.: November 1999); Internal Control Management and Evaluation Tool, 
GAO-01-1008G (Washington, D.C.: August 2001). 
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performance challenges. Appropriate and effective internal control is a 
key factor in helping agencies better achieve program results. 

The contract also requires the contractor to provide a quality assurance 
plan. The plan that was developed by the contractor describes the 
contractor's approach and strategy for ensuring the delivery of high-quality 
service. As part of the plan, the contractor is to conduct formal, biweekly 
internal performance review meetings to help with the identification and 
correction of performance deficiencies. These meetings are to be attended 
by contractor and USCIS officials, with contractor staff reporting on 
quality performance issues, and are to be in addition to the quarterly (now 
monthly) meetings discussed above. Minutes of the meetings are to 
identify action items, responsibilities, and solution time frames, and the 
minutes are to be published for USCIS review. However, a contractor 
official said that these meetings never took place. According to both 
contractor and USCIS officials, the quarterly meetings were used to 
discuss operations and performance and to focus senior management 
attention on any performance issues. 

Under FAR § 37.602-2, the government's quality assurance surveillance 
plans should include actions to help ensure that the contractor carries out 
its quality control obligations. By failing to ensure that the contractor held 
and documented performance review meetings as required by the 
contractor's quality assurance plan, USCIS did not meet its quality 
assurance obligations under FAR § 37.602-2 and GAO's internal control 
standards. In addition, USCIS's failure to obtain the contractor's written 
acknowledgment of USCIS-identified performance deficiencies did not 
meet the notification procedures established by the contract, 
documentation requirements of FAR § 46.104(c), and GAO's standards for 
internal control. 

In its comments on a draft of this report, DHS noted that the contract was 
administered by a component of the VA until April 20, 2005, and DHS said 
the VA was provided with documentation discussed at quarterly and 
monthly performance assessment meetings between USCIS and the 
contractor. According to DHS, the lack of a clear understanding between 
USCIS and the VA regarding their roles contributed to the fact that formal 
documentation and evaluations were not always properly maintained and 
formally transmitted to the contractor. DHS acknowledged that the agency 
procuring a service is ultimately responsible for the contract and, thus, 
USCIS should have clarified its and the VA's roles. 
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USCIS Has Used 
Results of Monitoring 
Efforts to Identify 
Opportunities to 
Improve Customer 
Service and Call Flow 
at All Call Centers 



USCIS used contractor performance data, including the results of surveys, 
call monitoring, and the mystery shopper program, to identify 
opportunities to improve customer service, including improving call- 
response times, help CSRs and IIOs better respond to customer inquiries, 
and manage the flow of calls into call centers. Following are examples of 
initiatives that USCIS recently implemented, or was planning to 
implement, as of April 2005. It is too early to assess the impact of these 
initiatives. 

• USCIS implemented "intelligent call routing" for Tier 1. With 

"intelligent call routing" in place since February 2004, telephone calls 
routed from the interactive voice response system to Tier 1 are now 
routed to the next available CSR, at any of the four Tier 1 contractor- 
operated call centers. Previously, telephone calls to Tier 1 were routed 
to the next available CSR in the call center that resided in the same 
geographic region of the country as the caller. USCIS officials said that 
by June 2005 they will also have implemented "intelligent call routing" 
for calls transferred from Tier 1 to the next available IIO at either of the 
two Tier 2 call centers. 



• USCIS implemented "overflow routing." USCIS started its "overflow 
routing" initiative in October 2004, enabling certain general call types, 
identified as "English Other" and "Spanish Other," to be routed directly 
from the interactive voice response system to Tier 2 USCIS-operated 
call centers, bypassing the Tier 1 contractor-operated call centers. 
Previously, all calls handled by IIOs at Tier 2 were first routed from the 
interactive voice response system to Tier 1, where CSRs then 
transferred the calls to Tier 2. USCIS officials said they expect the 
change will result in 1 to 5 percent of all calls being routed directly to 
Tier 2, which should help when Tier 1 CSRs cannot handle the call 
volume. 



• USCIS implemented interactive voice response system routing of 
certain telephone calls to USCIS service centers. USCIS changed its 
automated interactive voice response system in December 2004 so that 
certain types of customers' telephone calls — for example, certain 
issues concerning new permanent residents, cases already approved or 
denied, and pending cases — are now routed directly to USCIS service 
centers, bypassing CSRs at Tier 1. Previously, all customers' telephone 
calls that needed to be handled by USCIS service centers were routed 
by the interactive voice response system to Tier 1. Then, after talking 
with the customers, the CSRs referred the customers to the service 
centers (whose employees have access to case paperwork) via e-mail. 
CSRs were allowed to transfer customers' telephone calls to service 
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center personnel only when customers requested emergency and 
expedited handling of applications. 

• USCIS implemented a portfolio management system. Private attorneys, 
paralegals, and other representatives can use the USCIS Internet Web 
site to check the status of their clients' immigration cases using a 
USCIS receipt number. Under the system, USCIS also notifies the 
representatives via e-mail when a case status changes; for example, 
when actions are taken, such as the approval or denial of an 
application. As of April 2005, over 300,000 customers, attorneys, and 
other representatives had used this system. 

• USCIS said it is planning to implement a referral management system. 
Currently, Tier 1 CSRs send, via e-mail, service request referrals to 
USCIS service centers and local offices for customers who call wanting 
to change addresses, schedule and reschedule appointments at 
application support centers, order forms, and resolve problems. After a 
referral is made, NCSC does not know whether the service center or 
local office responded to the customer in a timely manner or even 
responded at all. To better monitor this process, USCIS plans to 
implement a referral management system, with such service request 
referrals placed in a database and assigned a tracking number. The 
system is to (1) determine the proper service center or local office to 
process the referral, (2) assign the case to an adjudicator, (3) update 
the case on a daily basis, and (4) report once a month on case status. 
The referral management system is planned to be accessible to 
customers on USCIS's Internet Web site so they can make and track 
their own service request referrals. In addition, customers without 
Internet access are to be able to call on the telephone and CSR's will 
access the USCIS Web site and create referrals for them. USCIS plans 
that the referral management system will be fully operational during 
the summer of 2005. 

• USCIS is planning a customer service portal on USCIS's Web site. 
USCIS has a long-term goal of giving customers Internet access to 
information contained in the "scripts" used by Tier 1 CSRs to answer 
customers' questions. USCIS plans to establish a customer service 
portal on the USCIS Internet Web site, providing access to the 
information. The goal is to let customers with Internet access look up 
information themselves without having to call NCSC on the telephone, 
navigate the interactive voice response system, and wait for CSR's to 
answer. USCIS had not set a time frame for implementing this 
initiative. 
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Conclusions Immigration call centers are a vital information referral source used 

millions of times by immigrants and other interested parties seeking to 
obtain needed documents, regulatory information, up-to-date status 
information on immigration-related benefits and applications, and other 
information. To ensure that it serves its customers effectively and 
efficiently, USCIS appropriately used a performance-based contract, but 
its failure to finalize all aspects of the performance requirements before 
the contract was awarded hampered its ability to exercise performance 
incentives in the contract. As a result, USCIS lost the opportunity during 
the life of the contract to help ensure that it received the maximum level of 
service from the contractor. 



In addition, USCIS did not meet standards promulgated by federal 
acquisition regulations, GAO, and the contract itself pertaining to 
documenting the contractor's performance between 2002 and 2004, and 
adequately documenting notification of the contractor when the 
government perceived deficiencies in its performance. Failure to generate 
adequate documentation could impair USCIS's ability to conduct future 
contract negotiations and to preserve a complete and reliable record of 
contract performance needed to ensure accountability. 



Recommendation for 
Executive Action 



To improve USCIS's efforts for evaluating contractor performance and 
encourage quality services at call centers, we recommend that the 
Secretary of Homeland Security require the Director of USCIS take the 
following two actions: (1) finalize contract terms related to specific 
performance measurement requirements before awarding new 
performance-based call center contracts; and (2) maintain readily 
available written records of performance assessments and performance 
evaluation meetings with the contractor. 



Agency Comments 
and Our Evaluation 



DHS and the contractor provided formal comments and technical 
comments on a draft of this report, which we have incorporated, as 
appropriate. In its formal comments, DHS generally agreed with our 
recommendations. DHS said the draft solicitation for the new contracts 
specifically identifies performance requirements that are non-negotiable. 
DHS further stated that, as recommended by GAO, written records of 
performance assessments and performance evaluation meetings will be 
maintained and readily available for review by all interested parties. In its 
formal comments, the contractor provided additional language to further 
clarify this report. The contractor said the report accurately summarizes 
the complex nature of CIS's call center program and several challenges 
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created by significant post-award changes to that program. DHS's and the 
contractor's formal comments are shown in appendixes VI and VII, 
respectively. 



We are sending copies to the Director of USCIS and other interested 
parties. We will also make copies available to others upon request. In 
addition, the report will be available at no charge on GAO's Web site at 
http://www.gao.gov. 

If you or your staff have any questions regarding this report, please 
contact me at 202-512-8777 orjonesp@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on the 
last page of this report. Key contributors to this report are listed in 
appendix VIII. 




Paul L. Jones, Director 

Homeland Security and Justice Issues 
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Appendix I: Scope and Methodology 



To determine what performance measures the U.S. Citizenship and 
Immigration Services (USCIS) established to monitor and evaluate the 
performance of contractor-operated call centers, we interviewed USCIS 
headquarters officials in Washington, D.C.; Tier 1 contractor officials in 
Arlington, Virginia, and at a contractor-operated call center; 1 and an 
official representing an independent consulting firm under contract to 
USCIS and located in Fairfax, Virginia. We also collected and analyzed 
pertinent USCIS and contractor documentation. We collected and 
analyzed information on the various types of monitoring and evaluation 
programs used by USCIS, including internal call monitoring, independent 
call monitoring, customer satisfaction surveys, the mystery shopper 
program, and telephone call data provided by a telecommunications 
vendor. 

To find out how USCIS used the performance measures to evaluate the 
contractor's performance, we interviewed USCIS headquarters officials in 
Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia. We 
also collected and analyzed pertinent USCIS and contractor 
documentation. 

To determine what actions, if any, USCIS took or planned to take to 
strengthen call center operations, we interviewed USCIS headquarters 
officials in Washington, D.C., and Tier 1 contractor officials in Arlington, 
Virginia, and at a contractor-operated call center. We also collected and 
analyzed pertinent USCIS and contractor documentation. 

We assessed the reliability of telephone call volume data provided to 
USCIS by a telecommunications vendor, as well as USCIS and contractor 
staffing data. To carry out our data reliability assessments, we 
(1) reviewed information about the data, systems that produced the data, 
and data quality control procedures, and (2) interviewed USCIS and 
contractor officials knowledgeable about the data as necessary. We 
determined that the call volume and staffing data were sufficiently reliable 
for the purposes of this report. 



lr The contractor considers the locations of its call centers to be proprietary information. 
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Appendix I: Scope and Methodology 



We conducted our work between May 2004 and May 2005 in accordance 
with generally accepted auditing standards. 
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Appendix II: Criteria and Methodology Used 
by Contractor to Measure Quality of Tier 1 
Calls Monitored 



Table 1 : Call Quality Monitoring or "Soft Skills" 


Criteria 


Scoring range 
(0-3) 


Section score 
(Percent) 


Establish rapport 




3 


11.5 


Maintain composure 




3 


12.0 


Conversational style 




3 


11.5 


Active listening 




3 


12.5 


Efficient call flow 




3 


12.5 


Opening 




3 


10.0 


Call on hold 




3 


10.0 


Call transferred 




3 


10.0 


Closing 




3 


10.0 


Percent compliant 






100% 


Source: USCIS contractor. 


Table 2: Accuracy of Information Provided 


Criteria 


Scoring range 
(0-3) 


Section score 
(Percent) 


Uses software tools appropriately 




3 


9.0 


Provides accurate response 




3 


28.0 


Provides complete response 




3 


27.0 


Provides USCIS content only 




3 


9.0 


Satisfies caller's needs 




3 


27.0 


Percent compliant 






100% 


Source: USCIS contractor. 


Table 3: Accuracy of Capturing Information 


Criteria 


Scoring range 
(0-3) 


Section score 
(Percent) 


Completes referral when 
appropriate 




3 


12.0 


Completes referral record correctly 




3 


28.0 


Captures caller's information 




3 


30.0 


Verifies caller's information 




3 


30.0 


Percent compliant 






100% 



Source: USCIS contractor. 
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Appendix III: Criteria and Methodology Used 
to Measure Customer Satisfaction with 
Customer Service Representatives 



USCIS and an independent consulting firm jointly developed a telephone 
survey to measure customer satisfaction with the three levels of NCSC call 
center service — interactive voice response system, Tier 1 CSRs, and Tier 2 
IIOs. To carry out the survey each month, representatives of the 
independent consulting firm call 375 randomly selected customers. To 
assess the customers' satisfaction with the CSRs, the representatives read 
several statements and ask questions for the customers to rate their 
experiences with CSRs. For the customer satisfaction performance 
measure required in the contract, USCIS collects and summarizes data on 
the customers' responses to the four statements below. The customers are 
asked to rate their agreement with each of the statements using a scale of 
1 to 7 (1 is strongly agree and 7 is strongly disagree). 1 

1. The representative seemed to fully understand my questions. 

2. The representative was polite. 

3. The representative did not rush me. 

4. The representative answered my questions promptly. 



lr The complete scale for the four statements is as follows: 1-Strongly agree; 2-Mostly agree; 
3-Somewhat agree, 4-Mixed, Neither agree nor disagree; 5-Somewhat disagree; 6-Mostly 
disagree; and 7-Strongly disagree. 
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Appendix IV: Criteria and Methodology Used 
by Independent Consulting Firm to Measure 
Quality of Calls Monitored 



An independent consulting firm scored CSRs on 23 separate quality 
assurance factors as follows. 



Service orientation 


Greeted customer 




Verified customer 




Established rapport 




Used customer name 




Expressed empathy 




Maintained composure 




Expressed commitment 




Offered additional assistance 




Terminated call appropriately 


Issue identification 


Obtained information 




Listed actively 


Issue resolution 


Took responsibility 




Provided thorough information 




Summarized actions 


Communications 


Projected enthusiastic tone 




Conveyed confidence 




Spoke with clarity 




Used appropriate language 



Call management Controlled call 

Used time efficiently 
Minimized "dead air" 
Extended hold courtesies 
Transferred call appropriately 
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U.S. Citizen Who Wants to Bring a Fiance to the United States 

Script 1.1 



You, the Customer/Shopper are: 

□ A U.S. citizen who wants to bring your fiance to the United States to get married to him/her here. Your fiance 

presently lives outside of the United States. You met in person a year ago. You have been married once before and got 
a divorce 4 years ago. 



Inquiry You Have for the Customer Service Representative (CSR): 

□ My fiance live:* in {country of choice) and 1 would like to bring him/her here !« get married. What do ! need to do? 



Questions the CSR Should Ask you: 

□ Arc you a U.S. citizen? (Yes) 

□ Have you or your fiance ever been married before? (Yes, i was married once before and got a divorce 4 years ago) 

□ Have you met your fiance in person within the last 2 years? { Yes. a year ago) 

□ Where is your fiance currently living? (Pick a country outside the U.S.) 



CSR Answers Should Have the Following Elements: 

□ Since you are a U.S. citizen, you can file a i'iance(e) petition on behalf of your fiancefc). 

□ In order for your fiance" to come to the U.S. to marry you, you will need to file the Form I-129F on your fiance's 
behalf, You will also need to include a Form G-325A, one for you and one for your fiance. 

□ You can cither download the Form I-129F and the Form G-325A from our Web site at www.iMtdft.gov or we can send 
it to you in the mail. 

□ In your petition you must prove that: 

o You have met your fiance(e) in person within the past 2 years (there are only very limited exceptions to this 

requirement); and 
o Both you and your fianee(c) are legally free to many: and 

o You and your fiancc(c) have already decided to marry one another. You simply want to have ihe ceremony 
conducted in the U.S.. and plan to marry within 90 days after s/he enters the U.S. 

o Since you indicated you were previously married, you will also need to submit copies of documents showing 
that your previous marriage was legally terminated. 

□ If CIS approves the Form J-129F, it will be sent to the U.S. Consulate/Embassy nearest 
your fianee{c). They will contact your fiance(e) so he/she can be interviewed for a 
fiancefe) visa, If your fiance(e) is granted the visa, he/she will be able to come to the 
U.S. for up to 90 days so that you can get married to each other. 

□ Hance(e) status automatically expires after 90 days. It cannot be extended for any 
reason. If you don't get married within 90 days after your fiance arrives, he/she must 
leave the U.S. or s/he will be in violation of his or her immigration status. 



CSR Bonus Information 

Bonus Information Should Have all of the Following Elements: 

□ When you get married within the 90 da>>. your fiance becomes your spou.se and can apply immediately for permanent 
resident statu 1 -. 

□ After \our fiance enters the United States and you get married, if you need more information about applying for 
permanent resident status, please see our web site at www.tiscis.ftoy or call us. 



Black - Customer/Mystery Shopper/Aon Scenario 

Blue - Customer/Mystery Shopper/Aon 

Purple - CSR/Pearson 

Green - Bonus Questions and Answers 



Source: USCIS. 
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Appendix VI: Comments from the 
Department of Homeland Security 



Department of Homeland Security 

Waahinguin. DC: 20528 

-j$Sk Homeland 
Security 



June 17, 2005 



Mr. Paul L. Jones 

Director, Homeland Security and Justice Issues 
U.S. Government Accountability Office 
44! G Street, NW 
Washington, DC 20548 

Dear Mr. Jones: 

RE: Draft Report GAO-05-526, Immigration Services: Better Contracting 
Practices Needed at Call Centers (GAO Job Code 4403 1 8) 

The Department of Homeland Security (DHS) appreciates the opportunity to review and 
comment on the Government Accountability Office's draft report. As the auditors noted, 
United States Citizenship and Immigration Services (USCIS) call centers are a vital 
information referral source used millions of times by immigrants and other interested 
parties seeking to obtain needed documents, regulatory information, status on 
immigration-related benefits and applications, and other information. The Department 
and USCIS appreciate that GAO recognized. USCIS' use of contractor performance data, 
including the results of surveys, call monitoring, and the mystery shopper program to 
improve customer service. Management and staff at USCIS are proud of the success of 
the call centers to date. 

GAO concluded that to improve efforts for evaluating contractor performance and 
encourage quality services at call centers, USCIS officials should (1) finalize contract 
terms related to specific performance measurement requirements before awarding new 
performance-based call center contracts; and (2) maintain readily available written 
records of performance assessments and performance evaluation meetings with the 
contractor. We generally agree with the GAO\s recommendations and offer comments 
on each recommendation. 

in order to provide better context, we request that GAO mention early in the body of the 
final report that the contract in question was awarded and administered by a component 
of the Veterans Administration for the Immigration and Naturalization Service (INS), 
which included what is now USCIS. The contract was awarded to Pearson Government 
Solutions (the vendor) in January 2002, fourteen months before the DHS was established. 
This information is not found in the body of the draft report aside from a footnote. 



Page 31 



GAO-05-526 Immigration Services 



Appendix VI: Comments from the Department 
of Homeland Security 



2 

With respect to recommendation one, performance measures were, in fact, detailed in the 
call center contract, The final methodology by which performance against the group of 
stipulated criteria would be weighed remained to be finalized. At the time of the award, 
ENS management believed it appropriate to let the winning vendor have some input into 
the mechanics of the methodology since this contract represented a transition to 
performance based contracting for call center operations. However, this approach 
resulted m subsequent discussions with the vendor about performance that delayed 
application of the incentive and disincentive clauses. 

USCIS* new solicitation specifically identifies six contract performance requirements 
that are non-negotiable. The solicitation fully delineates evaluation mechanics and 
methodology, and states the contractor will be evaluated for performance incentives or 
disincentives beginning 90 days following the operational date of contract services. The 
solicitation is currently with the DM5 Procurement Office for review and issuance. The 
GAO was provided a copy of the proposed solicitation in February 2005. 

The report acknowledges USCiS' plans in this regard, but then states on page 18 of the 
draft: "However, at the time of our review, CIS had not yet issued a solicitation for a 
new contract confirming its intention, nor was any other confirmation provided in 
writing.** USCIS officials disagree with the statement since the GAO had a copy of the 
proposed solicitation. Had GAO indicated it needed further written confirmation since 
the new solicitation has not yet been released, USCIS would have provided it 

GAO also recommended that USCIS maintain readily available written records of 
performance assessments and performance evaluation meetings with the contractor. Just 
as INS and subsequently USCIS found that it is important to finalize all performance 
criteria and evaluation methodologies before releasing a solicitation, its experience with 
this procurement equally demonstrates the importance of clearly identifying respective 
roles when one agency uses another as a service provider. As noted above, the contract 
GAO reviewed was administered by a component of the Veterans Administration, VA 
received a 1 percent contract fee to provide contract administration services, including, 
but not limited to, Statement of Work review and issuance; contract, negotiations; contract 
and performance documentation; and contract advisory services 

USCIS experience with the contract has shown mat roles were not sufficiently clan tied 
with VA, As GAO noted in the report, from the beginning INS and then USCIS 
conducted monthly and quarterly performance assessment meetings with the call center 
contractor. These assessments included a review of the contractors' performance in 
relationship to the metrics outlined in the Statement of Work. These materials were also 
provided to V A. The lack of a clear understanding between the two agencies contributed 
to the fact that formal documentation and evaluations were not always properly 
maintained and formally transmitted to the vendor We acknowledge the agency 
procuring the service is ultimately responsible for the contract and thus 1NS/USC1S 
should have clarified its roles with the VA, 
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By agreement of both VA and USCIS, on April 20, 2005, USCIS assumed the 
responsibility for administering this contract. Additionally, USCIS plans to administer 
the successor proewement now under development. As recommended by GAO, written 
records of performance assessments and performance evaluation meetings will be 
maintained and readily available for review by all interested parties. 

We are providing technical comments to your office under separate cover. 



Sincerely, 



Steven Pecinovsky 
Director 

Departmental G AO DIG Liaison Office 
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JOHN MCNAMAKA CURTIS 

PRESIDENT AND 

CHIEF EXECUTIVE OFFICER 



June 15, 2005 



PEARSON GOVERNMENT SOLUTIONS 

4250 N. FAIRFAX DRIVE 
SUITE 1200 

ARLINGTON,VA 22203 
TEI. (703) 284-5602 
FAX (703} 284-5662 



Mr Paul L. Jones 
Director 




Homeland Security and Justice 
U.S. Government Accountability Office 
441 G Street NW 
Washington, DC 20548 

Dear Mr. Jones: 

We appreciate the opportunity to comment on the GAO draft report (QAO-05-526) 
regarding the USC1S management of call center contracts. Overall, the report accurately 
summarizes the complex nature of CIS' call center program, and several challenges 
created by significant post-award changes to that program. However, we believe the 
report would benefit from additional clarification on the following points: 

Contract Changes Resulted in Challenging Negotiations 

The draft report explains that during a 16 month period following contract award, CIS 
and Pearson Government Solutions negotiated on how performance metrics would be 
scored. Pearson's objective during these negotiations was to define performance metrics 
that were reasonable and attainable given the significant and ongoing changes made to 
the contract following the award. In the end, although the negotiations were substantive 
they did not produce an agreement. 

CIS Metrics Do Not Reflect Changes 

CIS unilaterally decided to apply performance measurements to the contract that were 
unacceptable to Pearson. We objected for two reasons: the contract requires mutually 
agreed upon performance measurements; and CIS' performance measurements did not 
take into account changes to the work required by the contract. For example, as the 
report states, the number of scripts CSRs are required to use in responding to callers 
increased from 400 to over 2,300. However, CIS's performance measurements did not 
reflect this or other changes. As a result CIS determined that PGS did not meet 4 of the 7 
metrics for the fourth quarter of 2004, triggering a payment reduction. Although we 
object to this determination, we stand ready to work with CIS to establish fair and 
equitable financial incentives. 

Pearson Government Solutions manages citizen interaction programs for governments 
worldwide. We understand what it takes for contact center programs to be successful 
even with unexpected dynamic changes. As with all contracts, the contractor and 
government must work together to identify resources needed to respond to changes, agree 
on expected outcomes, and ensure that performance measurements and contract 
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modifications reflect unexpected changes. The CIS program has had challenges in this 
regard. However, despite these challenges, we have worked closely with CIS to provide 
high-quality service and will continue to do so. 

In closing, I want to commend your team's efforts to thoroughly assess and accurately 
summarize these complex issues. We appreciate the opportunity to present these 
comments on the draft report. Please do not hesitate to contact me or my staff if you 
have any questions. 




Sincerely, 



President and Chief Executive Officer 
Pearson Government Solutions 
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The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting its 
constitutional responsibilities and to help improve the performance and 
accountability of the federal government for the American people. GAO 
examines the use of public funds; evaluates federal programs and policies; 
and provides analyses, recommendations, and other assistance to help 
Congress make informed oversight, policy, and funding decisions. GAO's 
commitment to good government is reflected in its core values of 
accountability, integrity, and reliability. 



The fastest and easiest way to obtain copies of GAO documents at no cost 
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts 
newly released reports, testimony, and correspondence on its Web site. To 
have GAO e-mail you a list of newly posted products every afternoon, go 
to www.gao.gov and select "Subscribe to Updates." 

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. Orders 
should be sent to: 

U.S. Government Accountability Office 
441 G Street NW, Room LM 
Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

To Report Fraud, 
Waste, and Abuse in 
Federal Programs 



GAO's Mission 



Obtaining Copies of 
GAO Reports and 
Testimony 
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Web site: www.gao.gov/fraudnet/fraudnet.htm 
E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470 



Con ^rPSsi orml Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

V^UI Lgl t^lUl Ldl u g Govemment Accountability Office, 441 G Street NW, Room 7125 

Relations Washington, D.C. 20548 



PnhllP Affairs ^ au ^ Anderson, Managing Director, AndersonPl@gao.gov (202) 512-4800 

UU U ^U- 1 ^ 11 o U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 



PRINTED ON^/^ RECYCLED PAPER 



